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Preface  &  Acknowledgements 


Welcome  to  our  Tenth  Annual  Acquisition  Research  Symposium!  We  regret  that  this 
year  it  will  be  a  “paper  only”  event.  The  double  whammy  of  sequestration  and  a  continuing 
resolution,  with  the  attendant  restrictions  on  travel  and  conferences,  created  too  much 
uncertainty  to  properly  stage  the  event.  We  will  miss  the  dialogue  with  our  acquisition 
colleagues  and  the  opportunity  for  all  our  researchers  to  present  their  work.  However,  we 
intend  to  simulate  the  symposium  as  best  we  can,  and  these  Proceedings  present  an 
opportunity  for  the  papers  to  be  published  just  as  if  they  had  been  delivered.  In  any  case,  we 
will  have  a  rich  store  of  papers  to  draw  from  for  next  year’s  event  scheduled  for  May  14-15, 
2014! 


Despite  these  temporary  setbacks,  our  Acquisition  Research  Program  (ARP)  here  at 
the  Naval  Postgraduate  School  (NPS)  continues  at  a  normal  pace.  Since  the  ARP’s 
founding  in  2003,  over  1,200  original  research  reports  have  been  added  to  the  acquisition 
body  of  knowledge.  We  continue  to  add  to  that  library,  located  online  at 
www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  70  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  encourage  your  future  participation. 

Unfortunately,  what  will  be  missing  this  year  is  the  active  participation  and 
networking  that  has  been  the  hallmark  of  previous  symposia.  By  purposely  limiting 
attendance  to  350  people,  we  encourage  just  that.  This  forum  remains  unique  in  its  effort  to 
bring  scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  It  provides  the  opportunity  to  interact  with  many  top  DoD 
acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both  in  the  formal 
panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals,  breaks,  and  the 
day-ending  socials.  Many  of  our  researchers  use  these  occasions  to  establish  new  teaming 
arrangements  for  future  research  work.  Despite  the  fact  that  we  will  not  be  gathered 
together  to  reap  the  above-listed  benefits,  the  ARP  will  endeavor  to  stimulate  this  dialogue 
through  various  means  throughout  the  year  as  we  interact  with  our  researchers  and  DoD 
officials. 

Affordability  remains  a  major  focus  in  the  DoD  acquisition  world  and  will  no  doubt  get 
even  more  attention  as  the  sequestration  outcomes  unfold.  It  is  a  central  tenet  of  the  DoD’s 
Better  Buying  Power  initiatives,  which  continue  to  evolve  as  the  DoD  finds  which  of  them 
work  and  which  do  not.  This  suggests  that  research  with  a  focus  on  affordability  will  be  of 
great  interest  to  the  DoD  leadership  in  the  year  to  come.  Whether  you’re  a  practitioner  or 
scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 
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Abstract 

The  U.S.  Navy  transitioned  to  computer-based  training  (CBT)  in  A  and  C  schools  in  2003 
after  a  2001  Revolution  in  Training  report  claimed  that  the  Navy  would  realize  savings  in  cost 
and  training  time  without  negatively  affecting  the  quality  of  sailors  arriving  to  the  fleet. 
Anecdotal  evidence  from  ship  personnel  suggested  otherwise.  This  study  analyzes 
maintenance  data  for  the  AN/SGG-89(v)  sonar  system  to  determine  whether  the  transition  to 
CBT  contributed  to  increased  fleet  maintenance  costs. 

Government  studies  showed  that  the  conversion  to  CBT  was  not  the  sole  contributing  factor 
to  increased  fleet  maintenance  costs  or  degraded  fleet  material  readiness.  Changes  to  the 
Navy’s  training,  maintenance,  and  manning  programs  during  the  early  2000s  were  all 
contributing  factors.  If  the  conversion  to  CBT  were  to  have  an  effect  anywhere  in  the  Navy 
maintenance  system,  it  should  be  seen  in  maintenance  activities  where  sailors  were 
performing  maintenance  on  ships.  Our  analysis  revealed  that  the  average  cost  of  these 
activities  was  significantly  greater  after  CBT  was  implemented.  This  would  support  the 
anecdotal  evidence  that  CBT  was  impacting  the  quality  of  maintenance  on  ships. 

Introduction 

Traditionally,  the  majority  of  specialized  skills  training  (known  as  “A”  and  “C”  schools) 
in  the  Navy  has  taken  place  in  a  classroom  setting  with  instructors.  At  the  turn  of  the 
century,  Navy  leadership  became  concerned  that  current  training  programs  would  not 
adequately  meet  future  demands.  As  a  result,  the  chief  of  naval  operations  (CNO)  chartered 
an  Executive  Review  of  Navy  Training  (ERNT)  to  review  the  Navy  training  system  and 
recommend  solutions  to  improve  training  effectiveness  and  meet  future  training  demands. 

The  ERNT  group  noted  that  formal  schoolhouse  training  requires  a  large  investment 
in  facilities,  instructors,  and  laboratories  and  that  future  training  demand  would  outstrip  the 
number  of  billets  available  under  the  legacy  schoolhouse  system  (Executive  Review  of  Navy 
T raining  [ERNT],  2001 ).  They  suggested  that  the  use  of  new  training  technologies  could 
help  meet  that  demand  while  reducing  the  cost  of  training.  Motivated  by  these  findings,  the 
Navy  established  Task  Force  EXCEL  (Excellence  through  Commitment  to  Education  and 
Learning)  to  develop  a  continuum  of  lifelong  learning,  use  a  streamlined  funding  process 
and  a  single  training  authority,  create  a  Human  Performance  Systems  Model  (HPSM),  and 
link  training  and  acquisition  (Naval  Personnel  Development  Command,  2002). 
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Part  of  the  Navy’s  new  strategy  included  the  use  of  new  training  technologies  such 
as  distributed  learning,  computer-based  training  (CBT),  collaborative  learning,  and 
computer-mediated  learning.  The  Navy  claimed  that  the  introduction  of  CBT  would  reduce 
both  training  time  and  training  costs  without  reducing  the  quality  of  training  received  (ERNT, 
2001).  Accordingly,  CBT  was  introduced  full-time  into  the  training  pipeline  in  fiscal  year  (FY) 
2003. 


A  2009  Naval  Inspector  General  (IG)  Report,  Computer  Based  Training ,  reported 
that  the  introduction  of  CBT  did  reduce  training  time.  However,  sailors  arriving  to  the  fleet 
under  CBT  did  not  usually  meet  the  required  Knowledge,  Skills,  Abilities,  and  Tools  (KSATs) 
upon  reporting  on  board.  Because  of  this,  ships  had  to  take  the  time  to  train  sailors  up  to 
acceptable  standards  (Naval  Inspector  General,  2009).  This  suggests  that  while  initial 
training  costs  may  have  been  reduced  by  CBT,  the  overall  cost  of  operations  and 
maintenance,  including  on-the-job  training  (OJT),  may  have  increased. 

This  study  examines  the  impact  of  CBT  on  Navy  training  costs  as  well  as  operations 
and  maintenance  costs  before  and  after  the  implementation  of  CBT.  We  first  look  at 
Department  of  the  Navy  (DoN)  Budget  Reports  from  FY2000  through  FY2010  to  determine 
the  macro-level  impact  of  CBT  on  Navy  costs.  At  the  macro  level,  there  are  many  variables 
besides  CBT  that  could  contribute  to  changes  in  maintenance  costs,  including  the  Global 
War  on  Terror  (GWOT)  and  increased  operations  tempo  (OPTEMPO).  However,  it  is 
impractical  to  isolate  the  impact  of  CBT  on  Navy  maintenance  costs  at  the  macro  level. 
Instead,  it  is  necessary  to  look  at  the  impact  of  CBT  on  a  particular  system,  program,  or 
technology.  This  research  effort  focuses  on  a  single  system,  the  AN/SQQ-89(v)  sonar, 
collecting  data  at  a  level  of  detail  that  allows  for  the  control  of  the  various  variables  that 
might  impact  maintenance  costs. 

We  start  with  a  discussion  of  the  Navy’s  classroom  training  system,  the  Revolution  in 
Training  and  CBT,  followed  by  a  look  at  the  Navy  maintenance  process  and  changes  in 
manning  and  maintenance  policies  during  the  2000s.  Next,  we  focus  on  a  single  Navy 
system,  the  AN/SQQ-89(v)  sonar  system,  and  examine  how  the  conversion  to  CBT  might 
have  affected  maintenance  costs  in  that  system. 

Training 

Training  in  the  Navy  occurs  throughout  a  sailor’s  career.  After  completing  recruit 
training,  sailors  are  sent  to  specialized  skill  training  in  their  designated  job  specialty,  or 
rating.  In-rate  training  begins  in  A  school,  where  sailors  learn  the  particular  skills  specific  to 
their  job.  From  there,  a  sailor  can  receive  additional  training  in  C  school.  Once  a  sailor  is 
assigned  to  a  ship,  he  or  she  receives  training  for  collateral  duties  such  as  quarterdeck 
watches,  anti-terrorism/force  protection  watches,  weapons  handling,  and  the  at-sea  fire 
party.  Additionally,  sailors  can  expect  to  receive  general  military  training  in  topics  ranging 
from  electrical  safety  to  suicide  prevention. 

Traditional  Schoolhouse  Training 

Until  the  early  2000s,  in-rate  training  in  the  Navy  was  conducted  in  a  formal 
schoolhouse  setting,  where  instructors  delivering  the  training  are  subject  matter  experts 
(SMEs)  on  the  material  they  are  teaching  (ERNT,  2001).  Typically,  SMEs  come  from  the 
fleet  and  have  experience  working  on  the  equipment  they  are  teaching  about.  Training  is 
delivered  in  the  form  of  lectures,  and  instructors  are  able  to  supplement  the  lecture  material 
with  tips  and  anecdotes  from  their  career  experiences  (Naval  Inspector  General,  2009). 

In  addition  to  lectures,  sailors  can  reinforce  their  understanding  of  the  material 
through  hands-on  experience  in  a  laboratory  setting.  In  maintenance  courses,  students  are 
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able  to  work  on  the  exact  equipment  they  will  see  in  the  fleet,  and  instructors  are  able  to 
simulate  equipment  casualties  for  technicians  to  troubleshoot.  Instructors  are  able  to  tailor 
the  delivery  of  material  to  a  class  based  on  the  students’  levels  of  comprehension.  For 
example,  if  a  class  has  difficulty  understanding  a  particular  concept,  the  instructor  can 
choose  to  spend  more  time  in  the  lab  to  reinforce  what  is  learned  during  the  classroom 
portion. 

There  are  several  benefits  to  instructor-led  training  (ILT).  Since  a  single  instructor 
teaches  a  large  group  of  students,  group  learning  techniques  can  be  employed  that  would 
otherwise  be  unavailable  in  one-on-one  or  CBT  instruction.  The  formation  of  small  groups 
within  a  class  fosters  team-building  and  allows  students  to  help  and  teach  each  other. 
Compared  to  the  costs  of  software  development,  testing,  and  hardware  purchase,  ILT  is  in 
some  ways  more  cost  effective,  depending  on  class  size  and  length  of  use.  Additionally,  the 
controlled  classroom  environment  offers  fewer  distractions  than  CBT  or  distance  learning. 
Finally,  ILT  doesn’t  take  as  long  to  develop  as  CBT.  It  takes  approximately  34  hours  to 
develop  one  hour  of  ILT  (Chapman,  2007),  while  it  takes  approximately  220  hours  to 
develop  a  standard  e-learning  course  (Chapman,  2006). 

ILT  also  has  its  disadvantages.  Since  everyone  has  different  learning  capabilities, 
some  students  may  be  more  advanced  and  become  bored  while  waiting  for  slower  learners 
to  catch  up.  Conversely,  slow  learners  may  have  difficulty  keeping  up.  Depending  on  the 
size  and  duration  of  the  course,  ILT  may  be  more  expensive  than  CBT. 

Revolution  in  Training 

In  October  2000,  the  Executive  Review  of  Navy  Training  (ERNT)  group  was  charged 
with  providing  insights  on  how  to  improve  and  align  training  organizations,  leverage  civilian 
training  practices,  and  use  new  technologies  to  provide  a  continuum  of  training  for  sailors. 
The  24-member  group  was  comprised  of  military  and  civilian  personnel,  members  of 
academia,  research  institutions,  and  industry.  In  2001,  ERNT  released  their  report, 
Revolution  in  Training:  Executive  Review  of  Navy  Training  Final  Report. 

During  their  review,  the  ERNT  group  noted  that  the  demands  for  training  had 
increased.  At  the  macro  level,  the  training  demands  are  driven  by  the  Required  Operational 
Capabilities  and  Projected  Operating  Environments  (ROC/POE).  ROC/POE  is  a  tool  that  is 
used  to  determine  specific  warfighting  missions  for  each  ship.  Training  requirements  are 
derived  from  these  missions  and  are  then  used  to  determine  specific  training  requirements 
for  sailors. 

Changes  in  the  ROC/POE  lead  to  increased  ship  training  requirements  which  are 
passed  down  to  the  sailor  level.  The  ERNT  group  noted  that  the  finite  number  of  seats 
available  in  the  Navy  schoolhouses  was  not  able  to  support  the  increased  training  demands. 
Because  of  this,  there  were  gaps  in  the  types  of  training  that  current  and/or  potential  sailors 
needed  and  what  could  be  delivered. 

In  many  cases,  this  resulted  in  billets  which  could  not  be  filled  because  there  were  no 
sailors  with  the  required  training  to  fill  them.  During  the  1990s,  several  other  items 
contributed  to  the  lack  of  trained  sailors.  First,  the  pool  of  experienced  sailors  had 
decreased  due  to  drawdowns  and  retirements.  Second,  it  was  difficult  to  compete  for  trained 
personnel  in  a  healthy  U.S.  economy,  and  many  trained  sailors  were  leaving  for  jobs  in  the 
civilian  sector. 

The  ERNT  group  suggested  that  technology  and  the  science  of  learning  offered 
several  opportunities  to  improve  the  Navy  training  system  by  reducing  training  time  through 
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CBT  and  offering  distributed  learning  opportunities  that  could  be  executed  at  the  workplace. 
This  is  discussed  further  at  the  end  of  the  report. 

Computer-Based  Training 

Computer-based  training,  or  CBT,  is  defined  as  “individual  or  group  self-paced 
instruction  using  a  computer  as  the  primary  training  medium,  to  include  web-delivered  Navy 
E-Learning  (NEL)”  (Naval  Inspector  General,  2009,  p.  ii).  In  Navy  A  schools,  students  go 
through  learning  modules  on  a  personal  computer  at  their  own  pace.  When  students  are 
done  processing  the  information  presented  on  the  screen,  they  click  “next”  to  proceed  to  the 
next  piece  of  information.  There  are  usually  small  knowledge  assessments  throughout  the 
module,  followed  by  a  final  knowledge  assessment  at  the  end  of  the  module  (Naval 
Inspector  General,  2009,  p.  7). 

Because  the  learning  is  self-paced,  instructors  were  replaced  with  “facilitators.” 
Facilitators  are  not  necessarily  SMEs  in  the  subject  matter  being  delivered  in  the  CBT 
modules.  The  purpose  of  the  facilitator  is  “to  ensure  classroom  rules  are  followed,  assist  with 
computer-related  issues,  and  monitor  student  progress.  They  do  not  provide  reinforcement 
of  learning  objectives  or  enhance  retention  of  course  material.”  The  problem  with  replacing 
instructors  with  facilitators  is  that  students  cannot  go  to  a  facilitator  with  a  question  about 
subject  material,  removing  the  opportunity  to  teach  when  a  student  is  confused  (Naval 
Inspector  General,  2009). 

There  are  several  advantages  to  CBT.  The  learning  is  self-paced  and  if  the  course  is 
offered  as  distance  learning,  the  schedule  to  take  the  course  is  flexible.  Students  can 
complete  the  course  at  their  own  paces,  which  generally  shortens  training  time.  Since  there 
are  no  instructors  involved,  the  message  doesn’t  change  from  one  person  to  the  next 
(Dhanjal  &  Calis,  1999).  In  addition,  the  Navy  was  able  to  reduce  training  time  using  CBT, 
which  resulted  in  cost  savings  in  training  manpower  and  infrastructure,  as  noted  by  the  Navy 
IG  (2009)  and  the  GAO  (201 0). 

However,  the  use  of  CBT  raised  concerns  in  the  fleet  about  the  level  of  knowledge  of 
sailors  reporting  to  ships  from  A  schools.  The  inspector  general  (IG)  noted  that  sailors 
arriving  to  the  fleet  under  CBT  did  not  usually  meet  the  required  KSAT  standards  and  were 
unfamiliar  with  the  equipment  they  would  be  working  on  and  the  tools  they  would  need  to 
use.  Because  of  this,  ships  had  to  take  the  time  to  train  sailors  up  to  acceptable  standards. 

In  fleet  interviews,  some  commands  reported  that  qualification  time  was  nearly  double  what 
it  was  before  the  introduction  of  CBT  (Naval  Inspector  General,  2009).  The  GAO  reports  in 
2010  and  201 1  made  similar  observations  and  concluded  that  the  change  to  CBT  had  a 
negative  impact  on  readiness. 

The  Navy  IG  and  GAO  reports  found  that  while  the  Navy’s  use  of  CBT  resulted  in 
cost  and  training  time  savings,  the  quality  of  sailor  reporting  to  the  fleet  was  not  as  well 
prepared  as  ILT-trained  sailors  of  the  past.  The  result  is  that  poorly-trained  sailors  may  have 
contributed  to  declining  material  readiness  in  the  fleet.  The  next  section  of  this  study 
examines  Navy  maintenance  practices  and  highlights  the  findings  of  the  2010  Fleet  Review 
Panel  on  Surface  Force  Readiness  report. 

Maintenance 

Navy  maintenance  occurs  on  three  levels:  organizational  level  (O-level),  intermediate 
maintenance  (IM)  activities,  and  depot  level.  This  section  of  the  study  discusses  all  three 
maintenance  levels.  Additionally,  this  section  discusses  changes  made  to  the  maintenance 
process  in  2003  which  were  reported  on  in  the  2010  Fleet  Review  Panel  on  Surface  Force 
Readiness  (known  as  the  Balisle  Report  for  its  chairman,  Vice  Admiral  [VADM,  Retired] 
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Phillip  Balisle),  a  report  that  discussed  declining  fleet  readiness  as  a  result  of  changes  to 
training,  maintenance,  and  manning  policies  in  the  early  2000s. 

Shipboard  maintenance  begins  with  the  Planned  Maintenance  System  (PMS).  PMS 
is  governed  by  Naval  Sea  Systems  Command  (NAVSEA;  2003)  Instruction  4790. 8B,  Ship’s 
Maintenance  and  Material  Management  (3-M)  Manual.  The  instruction  outlines  the 
requirements  for  PMS  on  shipboard  systems  and  equipment.  The  purpose  of  PMS  is  to 
provide  ships  with  the  means  to  plan,  schedule,  and  perform  preventive  maintenance 
onboard  and  to  identify  potential  equipment  problems  before  the  equipment  fails. 

If  corrective  maintenance  is  required,  the  maintenance  is  reported,  scheduled,  and 
performed  through  O-level  shipboard  maintenance.  Ship  maintenance  actions  are  reported 
in  Navy  Visibility  and  Management  of  Operating  and  Support  Costs  (VAMOSC),  under  Unit 
Level  Consumption  and  Manhours — Organizational  Corrective  Maintenance. 

Intermediate  maintenance  (IM)  is  “normally  performed  by  Navy  personnel  onboard 
tenders,  repair  ships,  Shore  Intermediate  Maintenance  Activities  (SIMAs),  aircraft  carriers, 
and  fleet  support  bases”  (Naval  Sea  Systems  Command,  2003,  p.  1-5).  IM  jobs  are  deferred 
corrective  maintenance  jobs  that  are  beyond  the  capability  of  the  ship’s  force  and  are  sent 
off-ship  for  completion.  IM  is  tracked  in  Navy  VAMOSC  under  Maintenance — Intermediate. 

Depot-level  maintenance  “requires  major  overhaul  or  a  complete  rebuilding  of  parts, 
assemblies,  subassemblies,  and  end  items,  including  the  manufacturing  of  parts, 
modifications,  testing,  and  reclamation”  (Naval  Sea  Systems  Command,  2003,  p.  1-5).  Depot 
maintenance  is  reported  in  Navy  VAMOSC  under  Maintenance  and  Modernization — Depot, 
Other  Depot. 

In  2009,  VADM  (Ret.)  Phillip  Balisle  was  directed  to  conduct  a  Fleet  Review  Panel 
(FRP)  of  surface  force  material  readiness.  The  report  noted  that  4,052  billets  were  removed 
from  Navy  ships  from  2001-2009.  While  billets  were  removed  from  ships,  requirements  such 
as  maintenance,  damage  control  watches,  training,  and  in-port  duties  were  not  reduced 
(Balisle,  2010).  The  shortcomings  of  CBT  described  in  the  previous  section  exacerbated  the 
problems  experienced  with  manning  reductions  since  sailors  were  not  arriving  on  board  with 
the  right  KSATs.  The  result  was  undermanned  ships  with  poorly  trained  sailors  with  not 
enough  time  or  know-how  to  perform  routine  maintenance  actions. 

In  addition  to  reduced  fleet  manning,  shore  facilities  also  received  manning  cuts.  This 
means  that  maintenance  that  was  intended  for  intermediate  maintenance  activities  was 
pushed  back  to  ship  personnel,  which  were  undermanned  and  poorly  trained.  In  addition  to 
the  shrinking  shore  workforce,  the  amount  of  time  the  ships  were  available  was  shortened 
from  15  weeks  to  nine  weeks  (Balisle,  2010).  These  actions  resulted  in  equipment  being  out 
of  commission  for  longer  periods  of  time. 

Finally,  the  2010  Balisle  report  noted  that  changes  in  PMS  were  made  because  ships 
couldn’t  meet  maintenance  requirements  due  to  reduced  manning.  Maintenance 
requirements  were  either  eliminated  or  extended  in  periodicity.  The  intent  was  to  shift 
maintenance  requirements  to  shore  facilities,  but  since  manning  was  reduced  ashore,  many 
requirements  went  away  completely.  The  elimination  and  extension  of  maintenance 
requirements  can  lead  to  more  opportunities  for  equipment  to  become  inoperable,  resulting 
in  degraded  fleet  readiness  (Balisle,  2010). 

The  Navy  introduced  several  major  changes  to  training,  maintenance,  and  manning 
policies  during  the  early  part  of  the  2000-2010  decade.  The  Balisle  report  found  that  training 
was  a  factor,  but  certainly  not  the  only  factor,  that  led  to  degraded  fleet  readiness.  Manning 
reductions  would  have  led  to  cost  savings  in  the  military  personnel  budget,  but  the  impact  of 
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the  reductions  may  have  resulted  in  maintenance  cost  increases  in  future  budgets  due  to 
deferred  maintenance  actions,  thus  confounding  the  effect  of  CBT.  Similarly,  changes  in 
maintenance  policies  may  have  impacted  maintenance  costs  in  future  years.  At  a  macro 
level,  the  impact  of  CBT  is  impossible  to  tease  out  (see  Gibson,  2012,  for  an  examination  of 
Navy  training,  operations,  and  maintenance  budgets  between  2000  and  2012).  For  this 
reason,  we  decided  to  examine  one  system  in  particular,  the  AN/SQQ-89(v)  sonar  system, 
in  hopes  that  we  could  separate  the  two  factors. 

AN/SQQ-89(v)  Sonar  System 

To  examine  the  effect  of  CBT  on  rising  maintenance  costs,  this  study  will  focus  on 
the  operating  and  support  (O&S)  costs  of  a  single  Navy  system,  the  AN/SQQ-89(v)  sonar 
system,  and  look  at  how  the  conversion  to  CBT  affected  maintenance  costs  in  that  system. 
An  analysis  by  Gibson  (2012)  showed  that  manning  levels  for  sonar  technicians  did  not 
change  significantly  from  FY2000-FY2010,  effectively  eliminating  manning  as  a  contributor 
for  the  AN/SQQ-89  O&S  costs  and  focusing  the  study  on  training  and  maintenance. 

The  AN/SQQ-89(v)  surface  ship  Anti-Submarine  (ASW)  Warfare  combat  system 
(referred  to  as  “the  89”  in  the  rest  of  this  paper)  is  an  integrated  network  of  sonar  systems 
designed  to  search,  detect,  classify,  and  engage  ASW  threats.  The  system  is  currently 
installed  on  CG-47  class  cruisers,  DDG-51  class  destroyers,  and  FFG-7  class  frigates.  The 
89  uses  a  variety  of  sensors  that  can  transmit  (active)  and  receive  (passive)  acoustic  data  in 
order  to  detect  and  classify  threats.  Data  from  the  sensors  can  be  correlated  and  targets  can 
be  localized  using  Target  Motion  Analysis  (TMA)  to  generate  a  firing  solution  for  weapons 
systems  (Jane’s  Information  Group,  2010). 

The  89  system  consists  of  15  different  variants.  Variants  differ  based  on  the  sensors 
chosen  and  the  version  of  each  sensor.  In  this  report,  only  variants  2,  3,  4,  6,  7,  and  9  were 
studied.  These  variants  were  chosen  because  they  were  on  board  ships  prior  to  the 
introduction  of  CBT  into  the  sonar  training  pipeline  (2003)  and  remained  on  board  after  CBT 
was  introduced.  This  allows  for  analysis  of  ship-maintenance  trends  both  prior  to  and  after 
the  introduction  of  CBT.  A  list  of  ships  per  variant  is  given  in  Table  1 . 
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Table  1.  List  of  Ships  and  AN/SQQ-89(v)  System  Variants  Used  in  This  Study 


(V)2 

SHIP 

FOMEPORT 

(V)3 

SH*>  |  HOMEPORT 

(V)6 

SHIP 

HOMEPORT 

CG  55 

LEYTE  GULF 

Nt>rfcBc.  VA 

CG  56 

SAN  JACKTO  iNorbk.  VA 

CG  68 

AN30 

Norbk.  VA 

FFGS 

MCNRWr 

Mayport  R. 

CG  57 

LAkE  CHAtl  PLAINj  San  Lkego  CA 

CG  69 

VICKSBURG 

Mayport  FL 

FFG2S 

SOCNE 

Mayport  FL 

CG  S3 

PHLFPIfE  SEA  | Mayport  FL 

CG  70 

LAKE  ERE 

Pear  Harbor.  Hi 

FFG29 

STEPHEN  W  GROVES 

Mavport  FL 

CG  71 

CAPE  ST  GEORGE 

San  Dego.  CA 

FFG32 

JOHN  HALL 

Mayport  FL 

(V)4 

SHI> 

HOMEPORT 

:: 

VELLA  GULF 

Norbk.  VA 

FFG33 

JARRET 

San  Diego.  CA 

DDGS1 

arleighbufke 

Ntabk.  VA 

DDGS2 

BARRT 

NorOk.VA 

FFG36 

UNDERWOOD 

Mayport  FL 

DOG  S3 

JON  PAUL  JOKES 

San  Diego.  CA 

FFG3S 

CURTS 

San  Diego.  CA 

DDG  54 

CURTTS  WLBLR 

Yokcsufca.  Japan 

FFG39 

DOYLE 

Mayport  FL 

DOG  55 

STOUT 

Norbk.VA 

FFG40 

HALYBURTON 

Mayport  FL 

DDG  56 

JOHNS  MCCAIN 

Yokosuta.  Japan 

FFG41 

MCCLUSKY 

San  Diego.  CA 

DDG  57 

Mrrso-ER 

Nortik.  VA 

FFG42 

KLAKR1NG 

Mayport  FL 

DDG  58 

LA3O0N 

Ntxbk.VA 

FFG43 

THACH 

San  Oego.  CA 

DDG  59 

RUSSELL 

Pea'  Harbor.  Hi 

FFG45 

DEWERT 

Mayport  FL 

DOG  60 

PAUL  HAMLTON 

Pear  Harbor.  Hi 

FFG46 

RENTZ 

San  Dego.  CA 

DDG  61 

RAVAGE 

Nortik.  VA 

FFG47 

NICHOLAS 

Kbrtft.  VA 

DDG  53 

STETHEM 

Yokosuka.  Japan 

FFG43 

VANCEGRFT 

San  CVego.  CA 

(V)7 

SH*> 

HOMEPORT 

DDG  Si 

CARIEY 

Mayport  R. 

FFG49 

ROBERT G BRADLEY 

Mayport  FL 

CG  66 

HUE  CITY 

Mayport  R. 

DDG  55 

BENFOLD 

Sar  D^go.  CA 

FFG53 

HAWES 

Ftorfatk.  VA 

CG  67 

SHOH 

Yokes  Jra.  Japan 

DDG  56 

GONZALEZ 

Norbk.VA 

FFG55 

ELROD 

TSrbk.  VA 

DDG  67 

COLE 

Norbk,  VA 

FFG55 

SMPSGN 

Mayport  FL 

(V)9 

SH»> 

HOMEPORT 

DDG  55 

THE  SULLIVANS 

San  Dego.  CA 

FFG57 

REUBEN  JAMES 

r^ar  Hartor.  Hi 

FFG37 

CROMMELJN 

Pear  Harpor.  Hi 

DDG  25 

MLIUS 

SanDego.  CA 

FFCH 

SAMUEL  B  ROBERTS 

Mayport  FL 

FFGSO 

TAYLOR 

Mayport  FL 

DDGX) 

HOPPER 

Pear  Harbor.  H 

FFG59 

KAUFFMAN 

fbrbk.  VA 

FFG51 

GARY 

Sar  Dego.  CA 

DDG  71 

ROSS 

Norbk.  VA 

FFG60 

ROOKEY  M  CAVS 

Everett.  WA 

FFG52 

CARR 

Nortfc.  VA 

DDG  72 

MAHAN 

Norbk.  VA 

FFG61 

INGRAHAM 

Everett.  WA 

FFG54 

FORD 

Seres.  WA 

DDG  73 

CECATUR 

SanDego.  CA 

DDG  74 

MCFAUL 

Norbu.  VA 

DDG  75 

DONALD  COOK 

Norbx.  VA 

DDG  76 

HGGNS 

SanOego.CA 

DDG  77 

OKAfE 

Pear  Harbor.  Hi 

DDG  78 

PORTER 

Norbk.  VA 

All  sonar  technicians-surface  (STGs)  attend  STG  A  school.  At  A  school,  students 
learn  the  basic  principles  of  the  STG  rating  including  oceanography  and  principles  of  sound. 
Following  A  school,  STGs  are  sent  to  different  courses  depending  on  whether  they  are 
operators  or  operator/maintainers.  STGs  who  are  strictly  operators  are  sent  to  a  sonar 
operator  course,  where  they  learn  how  to  operate  the  specific  89  variant  of  the  ship  to  which 
they  will  be  sent.  Maintainers  are  sent  to  C  school,  where  they  learn  the  technical  skills 
required  to  maintain  the  equipment  they  will  work  on  upon  reporting  to  their  ship  (Navy 
Personnel  Command,  2012). 

CBT  was  introduced  full-time  into  the  training  pipeline  in  FY2003,  after  the 
recommendations  of  the  ERNT  report  (Naval  Inspector  General,  2009).  Data  were  not 
available  to  show  how  STG  course  lengths  were  affected  by  the  conversion  to  CBT.  The 
2009  Navy  IG  report  examined  the  course  lengths  of  22  A  and  C  schools  for  ILT  and  CBT 
and  found  that  on  average,  CBT  course  lengths  were  26%  shorter  than  ILT  course  lengths 
(Naval  Inspector  General,  2009).  This  study  focuses  on  FY1999  through  FY2010  to  capture 
data  prior  to  and  after  the  introduction  of  CBT.  Initially,  FY1995  through  FY1998  were  also 
considered,  but  there  were  not  enough  data  available  during  this  time  frame  for  most  data 
categories.  The  raw  data  provided  were  analyzed  to  reveal  relationships  between  selected 
data  sets. 

Program  Executive  Office  Integrated  Warfare  System  5  (PEO  IWS5)  provided  a  list 
of  ships  equipped  with  the  AN/SQQ-89(v)  sonar  system.  The  list  included  ship  class,  ship 
name,  hull  number,  homeport,  and  89  variant  number.  Only  ships  with  AN/SQQ-89(v) 
variants  on  board  both  before  and  after  implementation  of  CBT  were  considered.  The  initial 
list  provided  by  PEO  IWS5  included  all  ships  of  the  CG-47,  DD-963,  DDG-51,  and  FFG-7 
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classes.  To  narrow  the  ship  list  to  match  the  scope  of  our  study,  ships  were  removed  from 
the  data  set  if 

•  the  ship  was  decommissioned  during  the  FY1995-FY2006  time  frame, 

•  the  ship  received  a  variant  upgrade, 

•  the  ship  was  commissioned  FY2000  or  later,  or 

•  the  ship  was  outfitted  with  a  variant  introduced  after  FY2003. 

Using  these  criteria,  the  ship  list  was  reduced  to  68  ships.  VAMOSC  provided  O&S  cost 
data,  underway  steaming  days,  and  selected  non-cost  data  for  ships  equipped  with  the 
AN/SQQ-89  sonar  system  covering  FY1995  through  FY2010.  Cost  figures  were  given  in 
then-year  and  constant  FY201 1  dollars. 

In  addition  to  the  overall  89  system  data,  detailed  ship  data  were  available  for  the 
selected  ships.  Non-cost  data  included  number  of  personnel  trained,  maintenance 
manhours,  and  number  of  maintenance  actions.  The  data  were  used  to  calculate  average 
time  (in  manhours)  spent  per  maintenance  action.  These  numbers  were  calculated  to 
determine  training  and  maintenance  trends  pre-  and  post-CBT  (see  Figure  1).  For  instance, 
if  the  average  time  spent  per  maintenance  action  increased,  it  could  suggest  a  backlog  of 
maintenance  ora  lack  of  technical  competence  in  performing  a  maintenance  action.  Prior  to 
CBT,  manhours  spent  per  maintenance  action  were  trending  upward;  after  the  introduction 
of  CBT,  manhours  per  maintenance  action  remained  relatively  flat.  This  suggests  that 
manhours  per  maintenance  action  may  have  reacted  positively  to  the  conversion  to  CBT; 
however,  this  assumes  that  the  types  of  maintenance  actions  performed  remained  relatively 
constant. 


Single-factor  regression  analysis  was  performed  to  explore  the  relationship  between 
total  training  cost  and  the  O&S  variables  provided  by  Navy  VAMOSC.  Unit  Level 
Consumption,  IM,  Equipment  Rework,  and  Depot  Maintenance  were  selected  as  variables 
to  determine  whether  changes  in  training  costs  resulted  in  increased  maintenance  costs. 
The  selected  variables  represent  organizational-level,  intermediate,  and  depot-level 
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maintenance.  Unit  Level  Consumption  is  a  summation  of  Organizational  Repair  Parts, 
Replenishment  Spares,  and  Logistics  Center  (LOGCEN)  exchanges.  Equipment  Rework  is  a 
summation  of  contractor  and  government  Program  Office  rework  costs.  IM  is  a  summation  of 
afloat  and  ashore  IM  labor  costs  and  ashore  IM  materials  costs.  Depot  Maintenance  is  a 
summation  of  private  and  public  shipyard  depot  costs.  Training  (Total)  is  a  summation  of 
Program  Office  and  NETPDTC  training  costs.  The  results  are  shown  in  Table  2. 

Table  2.  Regression  Analysis — O&S  Components 


R2 

Unit  Level 

Consumption 

0.033 

Equipment  Rework 

0.002 

Intermediate 

Maintenance 

0.348 

Depot-Level 

Maintenance 

0.208 

In  this  case,  regression  analysis  shows  that  none  of  the  factors  selected  have  a 
strong  relationship  to  total  training  cost,  suggesting  that  if  maintenance  costs  are  related  to 
training  costs  in  the  STG  rating,  there  are  other  factors  not  identified  in  this  study  that  are 
having  an  effect.  It  is  interesting  to  look  at  the  relationship  of  IM  costs  and  training  dollars  in 
a  scatter  diagram  (Figure  2). 
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Figure  2.  Training  vs.  Intermediate  Maintenance  Scatter  Plot 


Figure  2  suggests  that  there  may  be  a  weak  relationship  between  training  dollars  and 
intermediate  maintenance  costs  and  that  this  warrants  further  investigation.  Specifically,  the 
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plot  suggests  that  as  training  costs  increase,  intermediate  maintenance  costs  decrease. 
This  would  support  the  hypothesis  that  when  less  money  is  spent  on  training  (as  a  result  of 
switching  to  CBT),  the  maintenance  costs  will  increase. 


Graphical  analysis  of  several  data  categories  indicated  noticeable  changes  after  the 
introduction  of  CBT.  For  example,  Labor  Ashore — Intermediate  Maintenance  Manhours 
showed  significant  change  (see  Figure  3). 
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Figure  3.  Labor  Ashore — Intermediate  Maintenance  Manhours 


The  figure  shows  the  number  of  manhours  spent  on  IM  for  selected  ships  from 
FY1995  through  FY2010.  Beginning  in  FY2004,  the  IM  manhours  increased  significantly  for 
the  selected  DDG-51  and  CG-47  class  ships.  While  this  may  be  partially  explained  by 
changes  to  Navy  maintenance  policy  described  earlier,  it  corroborates  the  evidence 
suggested  in  Figure  2,  namely  that  as  CBT  is  introduced  and  training  costs  decrease, 
intermediate  maintenance  hours  and  cost  increase.  This  trend  was  not  as  evident,  however, 
for  the  FFG-7  class.  This  may  be  explained  by  funding  of  the  ship  class,  since  many  of  these 
ships  belong  to  the  Naval  Reserve  Force  and  it  is  likely  that  their  funding  levels  did  not 
change  throughout  the  period  studied. 


Paired  t-tests  were  conducted  to  determine  whether  the  means  of  paired 
observations  (selected  ships  pre-  and  post-CBT)  were  different.  The  null  hypothesis  is  that 
there  is  no  significant  difference  in  the  means  before  and  after  the  introduction  of  CBT.  The 
alternate  hypothesis  is  that  there  is  a  significant  difference  between  the  means  due  to  CBT. 
A  negative  t-statistic  indicates  that  the  pre-CBT  mean  was  smaller,  while  a  positive  t-statistic 
indicates  that  the  post-CBT  mean  was  smaller.  Response  variables  used  in  this  study  were 
corrective  organizational  and  IM  actions,  organizational  parts  cost,  exchanges  LOGCEN 
cost,  manhours  organizational  labor,  and  labor  ashore  IM  manhours  (see  Table  3). 
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Table  3.  Paired  t-Test  Results 


Variable 

degrees 

freedom 

Number  of 
obs. 

Mean 

t  statistic 

p-value 

Corrective  org.  & 
IM  actions 

786 

before  CBT 

335 

61.43 

-6.61 

0.0000 

after  CBT 

458 

85.10 

Organizational 
parts  cost 

790 

before  CBT 

336 

6914.4 

-4.07 

0.0000 

after  CBT 

466 

9539.6 

Exchanges 
LOGCEN  cost 

563 

before  CBT 

238 

43752 

-6.30 

0.0000 

after  CBT 

328 

72178 

Manhours  org. 
labor 

779 

before  CBT 

335 

739.9 

-7.03 

0.0000 

after  CBT 

466 

1208.7 

Labor  ashore  IM 
manhours 

206 

before  CBT 

109 

107.51 

-5.17 

0.000 

after  CBT 

149 

357.17 

In  all  cases,  the  p-values  were  less  than  0.01,  indicating  a  significant  difference 
between  the  means  pre-  and  post-CBT.  This  suggests  that  the  introduction  of  CBT  had  a 
statistically  significant  impact  on  several  measures  of  maintenance  activity  and  cost. 


Interestingly,  the  number  of  maintenance  actions  (organizational  and  IM)  increased, 
even  though  changes  in  Navy  maintenance  policies  would  have  initially  led  to  fewer 
maintenance  actions.  Since  we  report  the  average  over  several  years,  it  is  possible  that  the 
expected  increase  in  future  maintenance  actions  was  part  of  the  observed  mean  after  CBT. 

It  is  also  likely  that  changes  in  operating  tempo  (due  to  the  GWOT)  had  a  significant  impact 
on  this  variable  as  well,  so  it  is  not  possible  to  isolate  the  effect  of  CBT  on  corrective 
maintenance  actions. 

Most  interesting  are  the  three  categories  related  to  maintenance  actions  performed 
by  sailors  at  the  ship  level:  organizational  parts  cost,  exchanges  LOGCEN  cost,  and 
manhours  organizational  labor.  If  the  conversion  to  CBT  were  to  have  an  effect  anywhere  in 
the  Navy  maintenance  system,  it  would  be  at  maintenance  activities  where  sailors  were 
performing  maintenance  on  ships.  This  data  would  support  the  anecdotal  evidence  provided 
by  ship  operators  that  CBT  training  would  also  impact  labor  ashore  IM  manhours  (in  IM 
facilities);  however,  a  confounding  variable  for  IM  is  that  several  shore-based  IM  facilities 
were  closed  and  shore-based  IM  billets  for  sailors  were  eliminated. 

Conclusion  and  Areas  for  Further  Research 

In  2001,  ERNT  released  its  report,  Revolution  in  Training:  Executive  Review  of  Navy 
Training  Final  Report,  which  led  to  a  major  overhaul  in  the  U.S.  Navy’s  training  practices, 
including  the  use  of  CBT  in  A  and  C  schools.  While  government  studies  of  the  Navy’s  CBT 
training  confirmed  that  the  transition  to  CBT  resulted  in  shorter  training  times  and  cost 
savings,  sailors  reporting  to  the  fleet  were  not  as  well  prepared  as  classroom-trained  sailors 
of  the  past,  and  extensive  OJT,  supervision,  and  assistance  in  performing  basic 
maintenance  tasks  were  required  to  bring  CBT-trained  sailors  up  to  speed. 

During  the  same  period  of  time  that  CBT  was  being  implemented,  the  U.S.  Navy 
reorganized  its  maintenance  program  and  reduced  total  manning  levels  on  ships,  even 
though  ship  requirements  did  not  change,  meaning  that  ships  had  to  do  more  work  with 
fewer  personnel.  Many  maintenance  requirements  were  deferred  or  eliminated  on  ships  with 
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the  expectation  that  shore  facilities  would  pick  up  the  slack,  but  shore  facilities  also 
experienced  manning  reductions.  As  a  result,  less  planned  maintenance  was  being 
performed  on  equipment,  which  increased  opportunities  for  equipment  failure  and 
decreased  fleet  material  readiness. 

This  study  looked  at  costs  from  a  systems  perspective,  considering  not  only  the  cost 
of  training  but  also  the  cost  of  maintenance.  We  asked  the  following  question:  If  sailors 
trained  with  CBT  had  lower  knowledge  and  skill  levels,  did  this  contribute  to  increased 
operations  and  maintenance  costs? 

Unfortunately,  there  were  too  many  confounding  variables  that  could  have  affected 
operation  and  maintenance  costs  during  this  period  of  time  to  draw  any  conclusions  about 
the  effect  of  CBT  on  maintenance  costs  from  the  Navy  level.  Instead,  we  focused  on  a 
single  Navy  system,  the  AN/SQQ-89(v)  sonar  system,  to  examine  the  effects  of  the 
conversion  to  CBT  on  maintenance. 

The  results  of  the  study  revealed  several  pieces  of  useful  information.  Regression 
analysis  indicates  a  weak  relationship  between  decreasing  in  training  costs  and  an 
increasing  in  IM  costs.  In  addition,  paired  t-tests  showed  that  the  conversion  to  CBT  may 
have  led  to  increases  in  corrective  organizational  and  IM  actions,  organizational  parts  cost, 
exchanges  LOGCEN  cost,  manhours  organizational  labor,  and  labor  ashore  IM  manhours. 
Of  particular  interest  were  results  for  manhours  organizational  labor,  organizational  parts 
cost,  and  exchanges  LOGCEN  cost,  all  associated  with  maintenance  performed  by  sailors 
at  the  unit  (ship)  level,  because  conversion  to  CBT  training  would  be  most  noticeable  at 
maintenance  activities  where  sailors  are  performing  the  maintenance. 

As  the  Navy  IG,  GAO,  and  Balisle  reports  suggest,  there  are  several  factors  that 
have  contributed  to  declines  in  fleet  readiness.  Most  notably,  the  simultaneous  combination 
of  changes  in  training,  maintenance,  and  manning  policies  appear  to  have  had  lasting 
negative  impacts,  including  rising  fleet  maintenance  costs.  The  data  analysis  performed  in 
this  study  shows  that  the  change  to  CBT  was  statistically  significant  when  compared  to 
several  maintenance  variables,  but  it  is  also  likely  that  changes  to  all  three  areas  (training, 
maintenance,  and  manning)  had  collective  negative  effects  which  go  much  further  than 
rising  maintenance  costs  and  actions.  It  is  clear  that  policy  changes  in  the  2000s  impacted 
fleet  readiness  in  a  negative  manner,  but  no  clear  conclusions  can  be  drawn  about  the 
specific  impact  of  CBT  on  total  system  cost  from  the  data  examined  in  this  study. 

Because  the  data  collected  can  be  characterized  as  panel  data,  statistical  analysis 
that  recognizes  the  panel  nature  of  the  data  will  be  performed  and  reported  in  another 
paper.  It  may  be  useful  to  study  the  impacts  of  the  conversion  to  CBT  on  other  Navy 
systems.  From  a  training  perspective,  the  lack  of  measures  of  effectiveness  for  training  may 
prove  frustrating  in  drawing  any  conclusions,  but  from  a  cost  perspective,  it  may  be  possible 
to  gain  further  insight  into  the  types  of  cost  most  affected  by  CBT. 
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